<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
 "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html>
<head>
 
 <title>papy - Google Code</title>
 <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
 
 <link rel="shortcut icon" href="http://www.gstatic.com/codesite/ph/images/defaultlogo.png"/>
 <link type="text/css" rel="stylesheet"
       href="./_static/custom.css" />
 <link type="text/css" rel="stylesheet"
       href="http://www.gstatic.com/codesite/ph/3770121226765569314/css/ph_core.css" />
 <link type="text/css" rel="stylesheet"
       href="http://www.gstatic.com/codesite/ph/3770121226765569314/css/ph_detail.css" />
 <link rel="stylesheet"  type="text/css"
       href="./_static/pygments.css" /> 
 
 <link type="application/atom+xml" rel="alternate"
 href="/feeds/p/papy/svnchanges/basic?path=/wiki/" />
 
 
<!--[if IE]>
 <link type="text/css" rel="stylesheet"
       href="http://code.google.com/hosting/css/d_ie.css" />
<![endif]-->

<script type="text/javascript">
      var DOCUMENTATION_OPTIONS = {
          URL_ROOT:    '',
          VERSION:     '1.0b1',
          COLLAPSE_MODINDEX: false,
          FILE_SUFFIX: '.html'
      };
    </script>
    <script type="text/javascript" src="./_static/jquery.js"></script>
    <script type="text/javascript" src="./_static/interface.js"></script>
    <script type="text/javascript" src="./_static/doctools.js"></script>
    <link rel="contents" title="Global table of contents" href="contents.html" />
    <link rel="index" title="Global index" href="genindex.html" />
    <link rel="search" title="Search" href="search.html" />
    <link rel="top" title="papy v1.0b1 documentation" href="index.html" />
    <link rel="next" title="Known Bugs" href="bugs.html" />
    <link rel="prev" title="Benchmark" href="benchmark.html" />
</head>
<body class="t6">
 <div id=gaia>
  <font size="-1">
 
 <a href="http://code.google.com/p/support/wiki/WhatsNew" style="color:#a03">What's new?</a>
 | <a href="http://code.google.com/p/support/">Help</a>
 
 </font> 

 </div>
 <table style="padding:0px; margin: 0px 0px -6px 0; width:100%" cellpadding=0 cellspacing=0>
 <tr>
 <td><a href="/"><img src="http://www.gstatic.com/codesite/ph/images/defaultlogo.png" alt=papy/></a></td>
 <td>
 
 <div id="pname">
 <a href="http://code.google.com/p/papy/" style="text-decoration:none; color:#000">papy</a>
 
 </div>
 <div id="psum">
 <i><a href="http://code.google.com/p/papy/" style="text-decoration:none; color:#000">papy v1.0b1 documentation</a></i>
 </div>
 
 </td>
 <td style="white-space:nowrap; text-align:right">
 
 <form action="http://code.google.com/hosting/search">
 <input size=30 name=q value=""/>
 <input type=submit name=projectsearch value="Search Projects" />
 <input type=submit name=websearch value="Search the Web" />
 </form>
 
 </tr>
 </table>


<table id="mt" cellspacing="0" cellpadding="0" width="100%" border="0">
 <tr>
 <th onclick="if (!cancelBubble) _go('http://code.google.com/p/papy/');">
 <div class="tab inactive">
 <div class="round4"></div>
 <div class="round2"></div>
 <div class="round1"></div>
 <div class="box-inner">
 <a onclick="cancelBubble=true;" href="http://code.google.com/p/papy/">Project&nbsp;Home</a>
 </div>
 </div>
 </th><td>&nbsp;&nbsp;</td>
 
 
 
 
 <th onclick="if (!cancelBubble) _go('http://code.google.com/p/papy/downloads/list');">
 <div class="tab inactive">
 <div class="round4"></div>
 <div class="round2"></div>
 <div class="round1"></div>
 <div class="box-inner">
 <a onclick="cancelBubble=true;" href="http://code.google.com/p/papy/downloads/list">Downloads</a>
 </div>
 </div>
 </th><td>&nbsp;&nbsp;</td>
 
 
 
 
 
 <th onclick="if (!cancelBubble) _go('http://code.google.com/p/papy/w/list');">
 <div class="tab inactive">
 <div class="round4"></div>
 <div class="round2"></div>
 <div class="round1"></div>
 <div class="box-inner">
 <a onclick="cancelBubble=true;" href="http://code.google.com/p/papy/w/list">Wiki</a>
 </div>
 </div>
 </th><td>&nbsp;&nbsp;</td>
 
 
 
 
 
 
 
 <th onclick="if (!cancelBubble) _go('http://code.google.com/p/papy/issues/list');">
 <div class="tab inactive">
 <div class="round4"></div>
 <div class="round2"></div>
 <div class="round1"></div>
 <div class="box-inner">
 <a onclick="cancelBubble=true;" href="http://code.google.com/p/papy/issues/list">Issues</a>
 </div>
 </div>
 </th><td>&nbsp;&nbsp;</td>
 
 
 
 
 
 
 <th onclick="if (!cancelBubble) _go('http://code.google.com/p/papy/source/list');">
 <div class="tab inactive">
 <div class="round4"></div>
 <div class="round2"></div>
 <div class="round1"></div>
 <div class="box-inner">
 <a onclick="cancelBubble=true;" href="http://code.google.com/p/papy/source/list">Source</a>
 </div>
 </div>
 </th><td>&nbsp;&nbsp;</td>
 
 
 
 <td width="100%">&nbsp;</td>
 </tr>
</table>
<table cellspacing="0" cellpadding="0" width="100%" align="center" border="0" class="st">
 <tr>
 
 
 
 <td>
 <div class="issueList">
<div class=isf>
 
 
 
 <a href="index.html">papy v1.0b1 documentation</a> |
 <span>
   <a href="genindex.html" title="General Index"
      accesskey="I">index</a>
 </span> |
 <span>
   <a href="modindex.html" title="Global Module Index"
      accesskey="M">modules</a>
 </span> |
 <span>
   <a href="bugs.html" title="Known Bugs"
      accesskey="N">next</a>
 </span> |
 <span>
   <a href="benchmark.html" title="Benchmark"
      accesskey="P">previous</a>
 </span> |
 <span class="inIssueList">
   <span>Search</span>
    <form action="search.html" method="GET" style="display:inline">
      <input size=30 name=q value=""/>
      <input type=submit name=projectsearch value="Search Documentation" />
      <input type="hidden" name="check_keywords" value="yes" />
      <input type="hidden" name="area" value="default" />
    </form>
 </span>

</div>
</div>

 </td>
 
 
 
 
 
 
 
 
 <td height=4 align=right valign=top class="bevel-right">
 <div class="round4"></div>
 <div class="round2"></div>
 <div class="round1"></div>
 </td>
 </tr>
</table>
<script>
 var cancelBubble = false;
 function _go(url) { document.location = url; }
</script>

<div id="maincol">
<!-- IE -->



<style>
 #downloadbox { 
 padding: 6px; 
 overflow: hidden;
 white-space: nowrap;
 text-overflow: ellipsis;
 }
 #downloadbox a {
 margin: 0 0 0 0;
 display: block;
 padding-left: 0px;
 }
 #owners a, #members a { white-space: nowrap; }
</style>

<div style="float:right; width:25em; margin:0 0 1em 2em">

 
 <div class=pmeta_bubble_bg>
 <div class="round4"></div>
 <div class="round2"></div>

 <div class="round1"></div>
 <div class="box-inner">
 <table class="pmeta" cellpadding=5>
 <tr><th>Previous:</th>
 <td><ul>
 
 <li>
   <a href="benchmark.html" title="previous chapter">Benchmark</a>
 </li>
 

 
 </ul></td>
 </tr>
 <tr><th>Next:</th>
 <td><ul>
 
 <li>
   <a href="bugs.html" title="next chapter">Known Bugs</a>
 </li>
 

 
 </ul></td>
 </tr>
 
 
 
 </table>

 </div>
 <div class="round1"></div>
 <div class="round2"></div>
 <div class="round4"></div>
 </div>
 
 
 
 <div style="background:#ddf8cc; margin-bottom: 5px;table-layout:fixed">

 <div class="round4"></div>

 <div class="round2"></div>
 <div class="round1"></div>
 <div class="box-inner">
 <div id=downloadbox>
  <b>Table of Contents</b><br/>
  <table class="pmeta toc" cellpadding=5><tr><td>
	<ul>
<li><a class="reference external" href="">Dictionary of terms and definitions</a><ul>
<li><a class="reference external" href="#map">map</a></li>
<li><a class="reference external" href="#imap">imap</a></li>
<li><a class="reference external" href="#id1"><em>IMap</em></a></li>
<li><a class="reference external" href="#worker-function">worker-function</a></li>
<li><a class="reference external" href="#worker-process-thread">worker-process/thread</a></li>
<li><a class="reference external" href="#worker">Worker</a></li>
<li><a class="reference external" href="#piper">Piper</a></li>
<li><a class="reference external" href="#dagger">Dagger</a></li>
<li><a class="reference external" href="#plumber">Plumber</a></li>
<li><a class="reference external" href="#stream">stream</a></li>
<li><a class="reference external" href="#input-stream">input stream</a></li>
<li><a class="reference external" href="#output-stream">output stream</a></li>
<li><a class="reference external" href="#input-item">input item</a></li>
<li><a class="reference external" href="#output-item">output item</a></li>
<li><a class="reference external" href="#input-piper">input <em>Piper</em></a></li>
<li><a class="reference external" href="#input-worker">input <em>Worker</em></a></li>
<li><a class="reference external" href="#output-piper">output <em>Piper</em></a></li>
<li><a class="reference external" href="#output-worker">output <em>Worker</em></a></li>
<li><a class="reference external" href="#input-node">input node</a></li>
<li><a class="reference external" href="#output-node">output node</a></li>
<li><a class="reference external" href="#lazy-evaluation">lazy evaluation</a></li>
<li><a class="reference external" href="#task">task</a></li>
<li><a class="reference external" href="#tasklet">tasklet</a></li>
<li><a class="reference external" href="#inbox">inbox</a></li>
<li><a class="reference external" href="#inbox-element">inbox element</a></li>
</ul>
</li>
</ul>

  </td></tr></table>

 
 </div>
 </div>
 <div class="round1"></div>
 <div class="round2"></div>
 <div class="round4"></div>
 </div>
 
 


 </div>
</div>

 <div style="padding:0 3em 1.2em 0">
   
     
  <div class="section" id="dictionary-of-terms-and-definitions">
<h1>Dictionary of terms and definitions<a class="headerlink" href="#dictionary-of-terms-and-definitions" title="Permalink to this headline">¶</a></h1>
<p>A dictionary of terms used within the documentation.</p>
<div class="section" id="map">
<h2>map<a class="headerlink" href="#map" title="Permalink to this headline">¶</a></h2>
<p>Higher-order map function. A function which evaluates another function on all
elements of the input collection.</p>
</div>
<div class="section" id="imap">
<h2>imap<a class="headerlink" href="#imap" title="Permalink to this headline">¶</a></h2>
<p>Iterated higher-order map function. A function which evaluates another function
on all elements of the input collection returning and evaluating the restuls
iteratively and lazily. <em>PaPy</em> depends on the imap implementation provided by
the standard Python imap from the itertoos module (itertools.imap) and the
custom <em>IMap</em>.</p>
</div>
<div class="section" id="id1">
<h2><em>IMap</em><a class="headerlink" href="#id1" title="Permalink to this headline">¶</a></h2>
<p>A parallel implementation of a multi-task imap function, which is used within
<em>PaPy</em>. It uses a pool of worker-threads or worker-processes and evaluates
functions in parallel either locally or remotely.</p>
</div>
<div class="section" id="worker-function">
<h2>worker-function<a class="headerlink" href="#worker-function" title="Permalink to this headline">¶</a></h2>
<p>A function with a standarized input written to be used by a <em>Worker</em> class
instance. All functionality of a <em>PaPy</em> pipeline has to be coded as
worker-functions.</p>
</div>
<div class="section" id="worker-process-thread">
<h2>worker-process/thread<a class="headerlink" href="#worker-process-thread" title="Permalink to this headline">¶</a></h2>
<p>A thread or process inside an IMap instance evaluating a tasklet remotely or
locally.
(related to: IMap, RPyC, thread, process)</p>
</div>
<div class="section" id="worker">
<h2>Worker<a class="headerlink" href="#worker" title="Permalink to this headline">¶</a></h2>
<p>An object-oriented wrapper for worker-functions, corresponds to
&#8220;function with partially applied arguments&#8221;</p>
</div>
<div class="section" id="piper">
<h2>Piper<a class="headerlink" href="#piper" title="Permalink to this headline">¶</a></h2>
<p>An object-oriented wrapper for Worker instances, corresponds to
&#8220;worker with defined mode of evaluation&#8221;
(related to: IMap, Worker)</p>
</div>
<div class="section" id="dagger">
<h2>Dagger<a class="headerlink" href="#dagger" title="Permalink to this headline">¶</a></h2>
<p>An directed acyclic graph (DAG) to store and connect piper instances.
(related to: Plumber, pipeline)</p>
</div>
<div class="section" id="plumber">
<h2>Plumber<a class="headerlink" href="#plumber" title="Permalink to this headline">¶</a></h2>
<p>A wrapper for the Dagger designed to run and interact with a running pipeline.
(related to: Dagger, pipeline)</p>
</div>
<div class="section" id="stream">
<h2>stream<a class="headerlink" href="#stream" title="Permalink to this headline">¶</a></h2>
<p>In Python terminology a stream is a file-like I/O object.</p>
</div>
<div class="section" id="input-stream">
<h2>input stream<a class="headerlink" href="#input-stream" title="Permalink to this headline">¶</a></h2>
<p>The input stream is the data which which enters a <em>PaPy</em> pipeline. The data is
assumed to be a collection of items expressed as a Python iterator (or any
object which has the next method).</p>
<p>Any sequence (e.g. a <tt class="docutils literal"><span class="pre">list</span></tt> or a <tt class="docutils literal"><span class="pre">tuple</span></tt>) can be made into an iterator using
the Python built-in <tt class="docutils literal"><span class="pre">iter</span></tt> function e.g:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="n">sample_sequence</span> <span class="o">=</span> <span class="p">[</span><span class="n">data_point1</span><span class="p">,</span> <span class="n">data_point2</span><span class="p">,</span> <span class="n">data_point3</span><span class="p">]</span>
<span class="n">sample_iterator</span> <span class="o">=</span> <span class="nb">iter</span><span class="p">(</span><span class="n">sample_sequence</span><span class="p">)</span>
</pre></div>
</div>
<p>Files are by default line-iterators i.e.:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="n">sample_file</span> <span class="o">=</span> <span class="nb">open</span><span class="p">(</span><span class="s">&#39;sample_file.txt&#39;</span><span class="p">)</span>
<span class="n">sample_file</span><span class="o">.</span><span class="n">next</span><span class="p">()</span> <span class="c"># returns the first line</span>
<span class="n">sample_file</span><span class="o">.</span><span class="n">next</span><span class="p">()</span> <span class="c"># returns the second line</span>
</pre></div>
</div>
</div>
<div class="section" id="output-stream">
<h2>output stream<a class="headerlink" href="#output-stream" title="Permalink to this headline">¶</a></h2>
<p>Input item saved and returned by an output <em>Piper</em>. By default the output
<em>Piper</em> should return a None for every input item, but save the result
persistently.</p>
</div>
<div class="section" id="input-item">
<h2>input item<a class="headerlink" href="#input-item" title="Permalink to this headline">¶</a></h2>
<p>A single element of the input strea.</p>
</div>
<div class="section" id="output-item">
<h2>output item<a class="headerlink" href="#output-item" title="Permalink to this headline">¶</a></h2>
<p>A single element of the returned or saved output stream.</p>
</div>
<div class="section" id="input-piper">
<h2>input <em>Piper</em><a class="headerlink" href="#input-piper" title="Permalink to this headline">¶</a></h2>
<p>A <em>Piper</em>, which is connected to a input stream (or multiple input streams) is a
input <em>Piper</em>. Such a <em>Piper</em> corresponds to a node in the graph which has no
upstream nodes within the <em>PaPy</em> pipeline or in other words has no outgoing edges
in the directed acyclic graph. An input <em>Piper</em> is an input node in the graph
representing the pipeline.</p>
</div>
<div class="section" id="input-worker">
<h2>input <em>Worker</em><a class="headerlink" href="#input-worker" title="Permalink to this headline">¶</a></h2>
<p>The <em>Worker</em> instance used to create the input <em>Piper</em>. The first <em>Worker</em>
function might depending on the type of the input stream have to deserialize the
data.</p>
</div>
<div class="section" id="output-piper">
<h2>output <em>Piper</em><a class="headerlink" href="#output-piper" title="Permalink to this headline">¶</a></h2>
<p>A <em>Piper</em>, which generates the output stream is a output <em>Piper</em>. A <em>PaPy</em>
pipeline might have multiple output <em>Pipers</em> in different places of the
pipeline. An output <em>Piper</em> corresponds to a node in the graph which has not
downstream nodes within the <em>PaPy</em> pipeline or in other words has no incoming
edges in the directed acyclic graph. An output <em>Piper</em> is an output node in the
graph representing the pipeline.</p>
</div>
<div class="section" id="output-worker">
<h2>output <em>Worker</em><a class="headerlink" href="#output-worker" title="Permalink to this headline">¶</a></h2>
<p>The <em>Worker</em> instance used to create the output <em>Piper</em>. The last worker function
should by convention return None and save the data persistently.</p>
</div>
<div class="section" id="input-node">
<h2>input node<a class="headerlink" href="#input-node" title="Permalink to this headline">¶</a></h2>
<p>A node, which has no outgoing edges. (note <em>not</em> incoming edges)</p>
</div>
<div class="section" id="output-node">
<h2>output node<a class="headerlink" href="#output-node" title="Permalink to this headline">¶</a></h2>
<p>A node, which has no incoming edges.  (note <em>not</em> outgoing edges)</p>
</div>
<div class="section" id="lazy-evaluation">
<h2>lazy evaluation<a class="headerlink" href="#lazy-evaluation" title="Permalink to this headline">¶</a></h2>
<p>Is the technique of delaying a computation until the result is required.</p>
</div>
<div class="section" id="task">
<h2>task<a class="headerlink" href="#task" title="Permalink to this headline">¶</a></h2>
<p>A task is an ordered tuple of objects added to the IMap instance it consists of:</p>
<blockquote>
<ul class="simple">
<li>a function, which will be evaulated on the input element-wise</li>
<li>an input (a list, tuple or any iterator object like a numpy array)</li>
<li>a tuple of arguments e.g. (arg1, arg2, arg3)</li>
<li>a dictionary of keyword arguments i.e. {&#8216;arg1&#8217;: value_1, &#8216;arg2&#8217;: value_2}</li>
</ul>
</blockquote>
<p>The optional arguments and keyworded arguments have to match the signature of the
function. The task will be iteratively split into jobs in the following way:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="n">tasklet</span> <span class="o">=</span> <span class="p">(</span><span class="n">func</span><span class="p">,</span> <span class="n">element_from_iterable</span><span class="p">,</span> <span class="n">arguments</span><span class="p">,</span> <span class="n">keyworded_arguments</span><span class="p">)</span>
</pre></div>
</div>
<p>(related to: IMap, Tasklet)</p>
</div>
<div class="section" id="tasklet">
<h2>tasklet<a class="headerlink" href="#tasklet" title="Permalink to this headline">¶</a></h2>
<p>A task is the unit of evaluation of a worker-thread/process within an IMap
instance. It is constructed from the task arguments with an input element the
tasklet:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="n">tasklet</span> <span class="o">=</span> <span class="p">(</span><span class="n">func</span><span class="p">,</span> <span class="n">element_from_iterable</span><span class="p">,</span> <span class="n">argumentns</span><span class="p">,</span> <span class="n">keyworded_arguments</span><span class="p">)</span>
</pre></div>
</div>
<p>Is evaluated as:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="n">result</span> <span class="o">=</span> <span class="n">func</span><span class="p">(</span><span class="n">element_from_iterable</span><span class="p">,</span> <span class="n">arguments</span><span class="p">,</span> <span class="n">keyworded_arguments</span><span class="p">)</span>
</pre></div>
</div>
</div>
<div class="section" id="inbox">
<h2>inbox<a class="headerlink" href="#inbox" title="Permalink to this headline">¶</a></h2>
<p>The first argument of any worker function. The elements of the function
correspond to the outputs of the upstream function in the <em>Worker</em> instance or
to outputs of other <em>Pipers</em>. These outputs are defined by the pipeline
topology. The contents of the inbox depend on a specific input item to the
pipeline. All other arguments of a worker function are predetermined.</p>
</div>
<div class="section" id="inbox-element">
<h2>inbox element<a class="headerlink" href="#inbox-element" title="Permalink to this headline">¶</a></h2>
<p>The number of elements in the inbox of a worker function is one if the function
is not the first function of a <em>Worker</em> instance or is equal to the number of
<em>Pipers</em> the worker function, <em>Worker</em> and <em>Piper</em> is connected to within a
<em>PaPy</em> pipeline.</p>
</div>
</div>


   
 
 
 </div>
<div id="footer" dir="ltr">
 
 <div class="text">
 
 &copy;2008 Google -
 <a href="http://code.google.com/">Code Home</a> -
 <a href="http://code.google.com/tos.html">Terms of Service</a> -
 <a href="http://www.google.com/privacy.html">Privacy Policy</a> -
  
 </body>
</html>
